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Abstract. We consider the three dimensional gravitational Vlasov Poisson sys- 
tem which is a canonical model in astrophysics to describe the dynamics of 
galactic clusters. A well known conjecture [6] is the stability of spherical mod- 
f~^ els which are nonincreasing radially symmetric steady states solutions. This 

T-H conjecture was proved at the linear level by several authors in the continuation 

^^ of the breakthrough work by Antonov [2] in 1961. In the previous work [29] . 

Cn we derived the stability of anisotropic models under spherically symmetric per- 

1— ( turbations using fundamental monotonicity properties of the Hamiltonian under 

, ^ suitable generalized symmetric rearrangements first observed in the physics lit- 

terature |34II12|H?1 [T]. In this work, we show how this approach combined with 
C^ a new generalized Antonov type coercivity property implies the orbital stability 

of spherical models under general perturbations. 
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< 

r^ 1. Introduction and main results 



s 



1.1. The gravitational Vlasov Poisson system. We consider the three dimen- 
sional gravitational Vlasov-Poisson system 



dtf + v- VJ - V(t>f ■ V^f = 0, (t, X, v)eR+xR''x 



^ \ f{t = 0,x,v) = fo(,x,v)>0, ^ ■ ^ 

O 



where, throughout this paper. 



P; Pfi^) = f{x,v)dv and (Pf{x) = --——* pj (1.2) 

^ JR3 47r|X| 

o 

y—{ are the density and the gravitational Poisson field associated to /. This nonlinear 

K^ transport equation is a well known model in astrophysics for the description of the 

• '-j mechanical state of a stellar system subject to its own gravity and the dynamics of 

rS galaxies, see for instance |6| 111). 

C^ The global Cauchy problem is solved in \63\ liTTl I5U] where unique global classical 
solutions /(t) in C^, the space of C^ compactly supported functions, are derived. 



Two fundamental properties of the nonlinear transport flow (1.1) are then first the 
preservation of the total Hamiltonian 

nf{t)) = lf \v\^f{t,x,v)dxdv-\ f \V(t>f{t,x)\^dx = 'H{fm, (1.3) 

andsecond the preservation of all the so-called Casimir functions: VG G (^""^([O, -|-oo),]E 
such that G(0) = 0, 

G{f{t,x,v))dxdv= f G{fo{x,v))dxdv. (1.4) 
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Equivalently, consider the distribution function associated to /: 

Vs>0, fif{s)=meas{{x,v) eR^ : f{x,v)> s} , (1.5) 



then (1.4) means the conservation law associated to nonhnear transportation: 

Vt>0, /i/(t)=^/o. (1.6) 

In this paper, we will deal with weak solutions in the natural energy space 

£ = {f>0 with / e L^ n L'^^iR^) and \v\^f G L\R^)} . (1.7) 

For all /o G £, ( |1.1[ ) admits a weak solution /(t), constructed for instance in |Hl22| 
I23j . whic h is also a renormalized solution, see [TJ E]. Moreover, this soluti on s till 



satisfies (1.4), belongs to C{[0,+oo), L^{R^)) and the energy conservation (1.3) is 
replaced by an inequality: 

vt>o, n{f{t))<n{fo). (1.8) 

1.2. Previous results. Jean's theorem [5j gives a complete classification of radially 



symmetric steady state solutions to ( 1.1 ). Recall that radial symmetry in our setting 



means /(x, v) = f{\x\, \v\,x ■ v). They are of the form 

Qix,v) = F{e,i) 
where e, i are respectively the microscopic energy and the kinetic momentum 

e{x,v)=^-Y + <pQix), i=\xAv\'^ (1.9) 

and are the only two invariants of the radially symmetric characteristic flow associ- 
ated to the transport operator t = v ■ V^ — V(/)q • V^. 

A canonical problem which has attracted a considerable amount of works both 
in the physical and the mathematical community is the question of the nonlinear 
stability of steady states models. The linear stability of all nonincreasing anisotropic 
models satisfying 

dF 

^<0 (1.10) 

is derived by Doremus, Baumann and Feix |10] (see also |13l ^^ ST] for related 
works), following the pioneering work by Antonov in the 60's [2l|3]. This analysis is 
based on some coercivity properties of the linearized Hamiltonian under constraints 



formally arising from the linearization of the Casimir conservation laws (1.4), see 
Lynden-Bell [34j . known as Antonov's coercivity property. 

At the nonlinear level, the full orbital stability in the natural energy space £ 
has been obtained for specific subclasses of steady states as a direct consequence 
of Lions' concentration compactness principle |3H 132) . see |46| [TH [161 113 EH El 
HOl [26l [271 l28l [38] . This powerful strategy however only applies to specific models 



which are global minimizers of the Hamiltonian (1.3) under at most two Casimir 
type conservation laws, see |271 [28] for a more complete introduction. 

A first attempt to treat the general case and use the full rigidity provided by 



the continuum of conservation laws (1.4) is proposed in |19] . |15| where the first 



result of stability against radially symmetric perturbations is obtained for the King 
model F{e) = (exp(eo — e) — 1),. The approach is based on Antonov's coercivity 
property and a direct linearization of the Hamiltonian near the King profile. 

We proposed in [29] a different approach based on fine monotonicity properties 
of the Hamiltonian under suitable generalized symmetric rearrangements as first 



ORBITAL STABILITY OF SPHERICAL GALACTIC MODELS 



observed in pioneering breakthrough works in the physics htterature, see in partic- 
ular Lynden-Bell [33], Gardner [l2j . Wiechen, Ziegler, Schindler |45| . Aly [1]. This 
approach avoids the dehcate step of hnearization of the Hamiltonian and reduces 
the stabihty problem for the full distribution function f to a minimization problem 
for a generalized energy involving the Poisson field (j)f only. The main outcome is 
the radial stability of nondecreasing anisotropic models, proved in |29) : 

Theorem 1.1 (Radial stability of nonincreasing anisotropic models, |29J). Let 
Q{x, v) = F{e, i) be a continuous, nonnegative compactly supported steady state so- 
lution to ( |1.1[ ). Assume that Q is nonincreasing in the following sense: there exists 
eo < such that F is C^ on O = {(e, £) G M x M+ : F{e, ^) > 0} C (-oo, eo) x M+ 
and 

dF 

^— < on O. 

oe 

Then Q is stable in the energy norm by radially symmetric perturbations, ie: for all 

M > 0, for all e > 0, there exists rj > such that given /o € C). radially symmetric 

with 

\\fo-Q\\L^<v, II/oI|l-<||Q||l-+m, \nfo)-'Hm<v, (i-n) 



the corresponding global strong solution f{t) to (1.1) satisfies: 



Vt>0, \\{l + \v\^){f{t)-Q)\\L^<e. (1.12) 

1.3. Statement of the result. Our aim in this paper is to extend the stability 
result of Theorem 1 1.1 1 to the full set of non radial perturbations. Here we recall that 
the radial problem enjoys an additional rigidity because for f{x.v) radially symmet- 



ric, the Casimir conservation laws (1.4) can be extended as follows: \/G{h,£) > 0, 
C^ withG{0,e) = 0, 

G{f{t,x,v),\x Av\'^)dxdv = G{fo{x,v),\x Av\'^)dxdv. (1-13) 

This additional conservation law is fundamental in the proof of Theorem |1.1[ and 
at the linear level, it is intimately connected to Antonov's coercivity property which 
is essentially equivalent to the coercivity of the Hessian of the Hamiltonian (|1.3|) 



under the full set of linearized constraints generated by (1.13) 



For the full non radial problem, (1.13) is lost. However, we claim that the strategy 
developped in |29| coupled with a new generalized Antonov coercivity property allows 
us to derive the classical conjecture of orbital stability of nonincreasing spherical 
models. 

Theorem 1.2 (Orbital stability of spherical models). Let Q be a continuous, non- 



negative, non zero, compactly supported steady solution to (1.1). Assume that Q is 
a nonincreasing spherical model in the following sense: there exists a continuous 
function F : M — t- M+ such that 

y{x,v)eR^ Q{x,v) = FO^ + (t>Q{x)Y (1.14) 

and there exists eo < such that F{e) = for e > cq, F is C^ on (— oo, eo) and 

F' <0 on (-oo,eo). (1.15) 



M. LEMOU, F. MEHATS, AND P. RAPHAEL 



Then Q is orbitally stable in the energy norm by the flow (1.1); for all M > 0, for 
all e > 0, there exists r] > such that, given fo€z£ with 

||/o-QIIli<^, nfo)<nQ) + v, II/oI|l-<||Q||l- + m, (i.ie) 

for any weak solution f{t) to ( |1.1[ ), there exists a translation shift z{t) such that 
Vi> 0, 

11(1 + \v\^){f{t,x,v) - Q{x - z{t),v))\\mRe) < e. (1.17) 



Comments on Theorem M. 21 

1. On the assumption on Q. Jean's theorem [5] ensures that the assumptions we 
make on Q are very general. Note that we allow F' to blow up on the boundary 
e — >■ Co which is known to happen for many standard models. We in particular 
extract from [6j two models of physical relevance which fit into our analysis: 

- The generalized polytropic models: 
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-^(e) = ^ ajj(eo-e)^, < qt < ^, aij>0. 

The King model: 

F{e) = (exp(eo — e) — 1) , for some cq < 0. 



2. Anisotropic models. Note that Theorem |1.2| deals with spherical models Q = F{e) 
while the full class of anisotropic models Q = F{e,i) is considered in Theorem |1.1[ 
Let us insist that the orbital stability of all anisotropic models with respect to 
non radial perturbations is not expected to hold in general (see [6j) and nonradial 
instability mechanisms may happen induced by the non trivial dependence on ki- 
netic momentum. We present a full non radial approach for spherical models only 
which is a canonical class, but which is likely not to be optimal. The derivation of 
sharp criterions of stability or instability for anisotropic models under non radial 
perturbations remains to be done. 



3. Quantitative bounds. The proof of Theorem 1.2 will rely on a compactness argu- 
ment, and one could ask for more quantitative bounds. Such bounds are available 
for the Poisson field and a consequence of our analysis is that for f £ £ satisfying 



(1.16), we can find Zf G M'^ such that 



nif) - n{Q) + Ufh^Wf* - QIIli > coWv^f - VM- - zf)\\ 



2 
L2 



for some universal constant cq > 0, see (4.4), where /* and Q* denote respectively 



the usual symmetric decreasing rearrangements of / and Q, as defined in Lemma 2.3 
The quantitative control of the full distribution function however seems to involve 
more subtle norms and would rely on weighted estimates for the bathtub principles 



see (2.25[). Such estimates were derived in the context of the incompressible 2D 



Euler in |36| 143] , but they seem to be more involved in our case due to the nonlinear 



structure of the generalized symmetric rearrangement that we consider, see (1.19). 
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1.4. Strategy of the proof. Let us give a brief insight into the strategy of the 



proof of Theorem 1.2 which extends the approach introduced in [29]. 



Step 1 . Monotonicity of the Hamiltonian under generahzed symmetric rearrange- 
ments. 

Let us define the Schwarz symmetrization of / as 

r(s)=inf{r>0 : /^/(t) < s}, (1.18) 



where Hf is defined by (1.5), which is the unique decreasing function on M+ with 

Given a potential i;^ in a suitable "Poisson field" class, we define the generalized 
symmetric nonincreasing rearrangement of / with respect to the microscopic energy 

2 



e = '-^ — h 4'{x) as the unique function of e which is equimeasurable to /, explicitely 



„6 1^ 



2 



nix,v) = foa4e{x,v)), a^e) = meas{{x , v) G R\ ^ + ,/.(x) < e}. (1.19) 



Any nonincreasing spherical steady state solution to (1.1) is a fixed point of this 
transformation when generated by its own Poisson field: 

g*'^e = Q. (1.20) 



Moreover, the Hamiltonian (1.3) enjoys a nonlinear monotonicity property which 



was first observed in the physics litterature, see in particular Aly jT]: 

n{f)>n{r't>f). (1.21) 

For perturbations which are equimeasurable to Q ie 

r = Q\ (1-22) 

we can more precisely lower bound the Hamiltonian by a functional which depends 
on the Poisson field only: 

^(/)-^(g)> J(0/)-J(0q) (1.23) 

where J can be interpreted as a generalized energy, |34j: 

Step 2. Coercivity of the Hessian: a Poincare inequality. 

We now linearize the functional J at </>q . The linear term drops thanks to the Euler- 



Lagrange equation (|1.20) and the Hessian takes the following remarkable form 



D^J{<pQ){h,h)= \yh\^ - \F'(e)\{h-UhYdxdv (1.24) 



where H, defined by (3.8), denotes after a suitable phase space change of variables 
the projection of h onto the functions which depend only on the microscopic energy 
e. A similar structure occured in |29| where the corresponding quadratic form was 

dF , 



\Vh\' 



-de^'^ 



{h - Yie/hydxdv (1.25) 



and where Hel corresponds to the projection onto functions which depend on (e, 



only (e and l being defined by (1.9)). The strict coercivity of the quadratic form 



M. LEMOU, F. MEHATS, AND P. RAPHAEL 



(1.25) was then equivalent to Antonov's stability result, but this statement is no 



longer sufficient in our setting as (1.25) is lower bounded by (1.24). 



We now claim the positivity of (|1.24|) for spherical models 

D^J{(l)Q){h,h) 



I \Vh\^ - I \F'{e)\{h-Iihfdxdv>^, (1.26) 

and in fact the quadratic form is coercive up t o the degeneracy induced by transla- 
tion invariancflj For this, we reinterpret (1.26) as a generalized Poincare inequality 
with sharp constant, and we claim that the classical approach developed by Hor- 
mander \2Qi\ |2TJ for the proof of sharp weighted L? Poincare inequalities: 



Jm.n 
under the convexity assumption 



|V/|^d^, / 



r? _ Jr^ f^^^ 



j^Nd^ 



W > 1 



:i.27) 



can be adapted to our setting. In particular, the non trivial convexity property 



of the steady state equation (1.20), see (3.48). 



(1.27) appears in the setting of (1.26) as a consequence of the non linear structure 



Step 3. Compactness up to translations. 

The outcome of Step 2 is the variational characterization of Q, (pq respectively as the 
locally unique (up to translation shift) minimizers of the respectively constrained 
and unconstrained minimization problems 

inf -H(/), \nij[4>). 
f*=Q* 

More precisely, we will show that i7(0) — J^{(J)q) controls the distance of (p to the 
manifold of translated Poisson fields (/'q(- + x), x £ M^, see Proposition 3.1 



From standard continuity arguments, the conservation law (1.6) and the inequal 



ity (1.8) ensure that Theorem 1.2 is now equivalent to the relative compactness in 



the energy space up to translation of generalized minimizing sequences: 
f*^Q* in L^ and Urn sup ?{{ f n) <n{Q). 



A slight improvement of the lower bound (1.23) implies first the relative compactness 
up to translations 

V(/'/„(- + x„)^V(/.q in L\R^). 
The strong convergence in the energy norm of the full distribution function now 



follows from a further use of the extra terms in the monotonicity property (1.21) 
which yields: 

{1 + \v\'^)\fn{x + Xn,v) — Q{x,v)\dxdv ^ as n — )■ +00 



and enables to conclude the proof of Theorem |1.2| 

The paper is organized as follows. In section 2, we show how a suitable phase- 
space symmetrization allows to reduce the study of the Hamiltonian Ti to the study 
of a functional J' which depends on the Poisson field (j)f only. In section [Sl we 
show that (j)Q is a local minimizer of this new functional and that J^{4>) — j\(t>Q) 
controls the distance of (/> to the manifold of translated functions (J)q{- + z), z £ M^, 



see Proposition 3.6 for precise statements 
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Proposition 3.1 In section 4, a sharp use of the monotonicity properties for both 



functionals % and J yields the compactness of the whole minimizing distribution 



functions. The proof of Theorem 1.2 then follows in section 5. 
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2. Reduction to a functional of the gravitational potential 

In this section, we introduce the notion of rearrangement with respect to a given 
Poisson type field, and show the monotonicity of the Hamiltonian under the cor- 
responding transformation which allows to compare the minimization problem of 
T-L{f) under the constraint /* = Q* to an unconstrained minimization problem on 
the Poisson field (pf only. Our approach extends the one we developed in [29J to 
the case of non radial potentials, and most arguments are in fact simplified by the 
absence of kinetic momentum. 

2.1. Properties of Poisson fields. Let us start with defining a suitable class of 
"Poisson type" potentials: 

X =\(j)(^ C{W^) such that < 0, lim 0(x) =0, V(/> G L^(M^) and m(0) > 

where 

m(</.) := inf {I + \x\)\4){x)\ (2.1) 



Notice that (2.1) implies: 

V0,0eA', VA>0, m{(t) + 4))>m{(l3) + m{4,), m{X(j)) = Xm{ct>), (2.2) 

and thus X is convex. Moreover, there holds: 
Lemma 2.1 (Properties of Poisson fields). Let f G £ nonzero and (pj be its Poisson 



field given by (1.2), then (pf € X 



Proof. Let f ^ £, nonzero. From standard interpolation estimates, pf E L^'^ Pi L^. 
Hence, by elliptic regularity, (j)f G W^^^^^, V(pf G L'^{R^) and (pf G C°'^ by Sobolev 



embedding. Also 0/ < and (pfix) ^ as |x| — t- -|-oo from (1.2). In particular, (pf 
attains its infimum on M^ with 

— oo < uiiiKpf < 0. 

It remains to show that m{(p) > which follows from the existence of C/ > such 
that: 

VxeR^ (j)(x)< ^. (2.3) 

^ ' ~ I + \x\ ^ ' 

Indeed, pick R > such that 

/ Pfiy)dy>Mh^>0, 

J\y\<R ^ 
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and estimate for Ixl > R: 



-m = I .r^^dy > / ,r^;4^dy > 



47r\x-y\ Jiyi^ji47r{\x\+R) 8Tr{\x\+R)' 



which yields (2.3). The proof of Lemma 2.1 is complete. D 



Let us now associate to </> G Af the following Jacobian function: 

Lemma 2.2 (Properties of the Jacobian a^). Let (j) ^ X . We define the Jacobian 
function a^ : WL — t- M+ as: 

Ve < 0, a(f,{e) = meas < (x, t>) G M : — — h (j){x) < e 

Then: 

(i) There holds the explicit formula: 

Ve < 0, a^{e) = ^^ / (e - ^{x)f_/^ dx. (2.4) 



In particular, a0(e) = for all e < raincj); 
(ii) 00 is C^ on (— oo,0) and is a strictly increasing C^ diffeomorphism from 
[mine/), 0) onto M4.. 

Proof. Let us prove (i). We have the inclusion 

I {x,v) e M^ : —^ + (/>(x) < e ^ C {{x,v) G M^ : (/)(x) < e and |i;p < 2(e -mine/))} 

Let e < 0. Since (p is continuous and goes to zero at the infinity, the set in the 



right-hand side is bounded in MP, thus a(^(e) < +00. The formula (2.4) now follows 
after passing to the spherical coordinates in velocity. We now prove (ii). Since, for 
all e < 0, the set |x G M^ : (/'(x) < e} is bounded, we may apply the dominated 
convergence theorem and get the continuity and differentiability of o^ on M^l , with 

a'Je) = 47r\/2 / (e - (l)ix))'^/^ dx. (2.5) 

Hence a'- is nonnegative and clearly continuous. Moreover, if a'Je) = then e — 
0(x) < for all X G M^, which means that e < miiKp. Therefore, if e > mint/), then 
o^(e) > 0. It remains to prove that lim 00(e) = +00. Since (j) £ X, we have 

aJe) > C ( e H —\ ] dx —^ +00 as e — ;> 0, 

from f^3 /-111 ^^3/2 = +00, m{(j)) > and the monotone convergence theorem. This 
concludes the proof of Lemma |2.2[ D 



2.2. Rearrangement with respect to the microscopic energy. We introduce 
in this section the generalized rearrangement of / with respect to a Poisson field G 
X. Let us start with recalling standard properties of the Schwarz symmetrization. 
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Lemma 2.3 (Schwarz syminetrization or radial rearrangement). Let f G L^nL°°, 

then the Schwarz symmetrization f* of f is the unique nonincreasing function on 
M4. such that f and f* have the same distribution function: 

Vs > 0, ^j{s) = fif*{s) 

with Hf given by (1.5) and /ij* defined analogously^ Equivalently, f* is the pseudo- 



inverse of fij: 

Vt > 0, f*{t) = inf {s > : fif{s) < t} . 
The following properties hold: 

(i) f* £ L\n L°° with 

r(0) = ll/IU-, Supp(r) C [0,meas(Supp(/))]; 
(ii) for all 13 G CHK+,K+) with /3(0) = 0, 

P{f*{t))dt= [ /3{fix,v))dxdv. (2.6) 

Observe that the above definition of /* is equivalent to 

Vt > 0, f*{t) = sup {s > : /i/(s) > t} , 

with the convention that /*(t) = when the set {s > : /Uj(,s) > t} is empty. 
Note also that if / is continuous then /* is continuous [42j. In particular, Q* is 
continuous. 

Given </> G X, we now define the rearrangement of / with respect to the micro- 
scopic energy '-^ — h (/>(x) as follows: 

Lemma 2.4 (Symmetric rearrangement with respect to a microscopic energy). Let 
f £ £ and let (j) £ X. Let f* be the Schwarz rearrangement in R^ given by Lemma 



(2.7) 



(i) Z**^ is equimeasurable with f, i.e. 

f*'^£Eq{f) = {g£LlnL^ with fXf = fig}. (2.8) 

(ii) Z**^ belongs to the energy space, i.e. f*'f' G £ with 

'f*^dxdv<C\\Vn%'\\f\\'l'\\f\f/l. (2.9) 



2.3 We define the function 






f*^{x,v) = < 


' /*(°^(? + '^(^))) 


^f 


^ + Hx) < 

lyl2 







if 


^-j- + (Pix)>0 


on M.^, where a^ is defi 


ned by (2. An. Then: 







\v\^ 



2 



Proof. Let us prove (i). The equimeasurability of / and f*"^ relies on the following 
elementary change of variable formula: let two nonnegative function a G C'^(M) n 
L'^iR) and 7 G Li(M+), then 



Ji^+9i(x)<o V 2 J \ \ 2 



bp \ / /Iwp \\ 

a [ -^^ + (j){x) 1 7 ( a</. ( -^ + 4>{x) 1 1 dxdv 

/•+00 





a{e)-f{a^{e))a'^{e)de = a (^a^^{s)j 'y{s)ds. (2.10) 



through the one dimensional Lebesgue measure 
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To obtain the first equality in (2.10), we pass to the spherical coordinates in velocity 



li = |z;| and perform the change of variable e = ^ + 4>{x) in the integral of u: 



-- 47r\/2 dx a(e)7(a<^(e)) (e - (/'(x))^/^ de. 

JR3 J(f,{x) 

47rv2 / a{e)j{a(j){e))de / (e — 0(x))_^ dx. 

J mint/. J]R3 



We conclude thanks to the formula (2.5) of a'l. The second equality comes after the 



change of variable s = a^{e). Recall from Lemma 2.2 that a^ is a C^ diffeomorphism 
from [min cp, 0) onto 



Let /3 G C^(M+,M+) such that /3(0) = 0. From (2.10) and the definition (2.7), we 



get 



+ 00 



I3(f*^{x,v))dxdv= I P{f*{s))ds= I P{f{x,v))dxdv, 



where we use (2.6). This proves that f*''' G Eq(/). 



Let us now prove (ii). From the equimeasurability of / and f*'^, we already 
deduce that 

ii/iiLi = iirtLi, ii/iu^ = iir^iiL^. (2.11) 

Moreover, we have 



-f'f'dxdv 



+ (t){x) f't'dxdv 



{x)f^dxdv 



< 



ix)r'^dxdv<mL^\\r\\Li<+oo, 



where we used (2.7). More precisely: 



f \}!l_f*c^^^^^ < - f (l)ix)f^dxdv= I Vcp-Vcpf,^dx 

< c\\vHLA\\v?r^\\^l.%rHT\\f*'^\\^i^ 

where we used the Cauchy-Schwarz inequality and the following standard interpo- 
lation inequality: for all (7 G <f^, 

IIV7J, l|2 ^-^111 |2 ||1/2|| ||7/6|| ,,1/3 (n^n\ 

||V^g||^2 < C\\\v\ g\\l^ \\g\\l^ llffll/oo, (2.12) 

and (2.9) follows. This concludes the proof of Lemma 2.4 D 



We end this subsection with an elementary lemma which will be useful in the 
sequel. 

Lemma 2.5 (Pseudo inverse of f* oa^). Let f € £, nonzero, and (j) ^ X . We define 
the pseudo inverse of f* o a,^ for s G (0, ||/||x,oo) as: 

{f*oa^rHs) = sup{ee[mm<p,0): f*oa^{e)>s}. (2.13) 

Then (/* o a^)^^ is a nonincreasing function from (0, ||/||z,oo) to [min(/),0) and for 
all {x,v) G M^ and s G (0, ||/||l°o); 



/*'/'(x, v)>s^^^ + ^{x) < if* o a^r\s), 



(2.14) 
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r'^{x,v)<s 



+ Hx)>iroa^)-\s), 



(2.15) 



where f*'^ is defined by (2.7). 

Proof. Let s G (0,11/11 Lcx,), then from /*(0) 
Lemma 12.21 



L°°, f*{t) — )• as f — )• +00 and 



{e G [mint/), 0) : /* o a0(e) > s} is not empty 



(2.16) 



and (/* o a^) ^(s) defined by (2.13) is strictly negative. The monotonicity of (/* 



o^) ^ follows from the monotonicity of /* and a^. Assume that f*^[x,v) > s, 



\vl 



then from the definition (2.7), we have min0 < -L^ — h (f){x) < 0. We also have 
/* o o^</)( 2 — '" '^(^)) ^ '^' therefore '-^ — h (j){x) < (/* o a^)~^{s) from the definition 



(2.13). This proves (2.14). Assume now that f*'f'{x,v) < s. Then, for all e G {e G 



[min0, 0) : /* o a0(e) > s} which is a non empty set, we have '-^ — \- (j){x) > e, and 



W 



(2.15) is proved. 



D 



2.3. Spherical models are fixed points of the generalized rearrangement. 



We now reinterpret the assumptions on Q in Theorem 1.2 and claim that spherical 
models are fixed points of the / — )• f*'^l transformatioij^ 

Lemma 2.6 {Q is a fixed point of the f*'''! rearrangement). Let Q be a radially 
symmetric spherical models as in the assumptions of Theorem\1.2\ Then we have 



F{e) = Q* oa^^ie), Ve G [(/>q(0),0), and Q*'I'Q = Q onR^. (2.17) 

Proof. Observe first that, since the boundary of {Q(x,v) > 0} is the level set 

2 — '" '/'q(^) ~ ^0) we have /Uq(0) = meas (Supp((5)). From the equimeasurability 
of Q and Q* , we have 

fiQ{F{e)) = meas <^ (x, t;) G 

= meas{s G M^, Q*{s) > F(e)}, 
for all e < cq. Since F is strictly decreasing on (— oo, eo], this is equivalent to 

fiQ{F{e)) = a^Q{e)= meas {s e Rl, Q*{s)>F{e)}, Ve < cq. (2.18) 

In particular a0„(eo) = meas(Supp((5)) > 0, which implies that (J)q{0) < cq. From 



\v\ 

T 



==6, f(^ + <Pq{x)] >F{e) 



(2.18) and the invertibility of both continuous functions F and a^p^ on [(f>Q(0), cq], we 
deduce that ^q is continuous and one-to-one from [0, -F((/>q(0))] to [O,a0g(eo)]. In 
particular, Q* is the inverse of fiQ on this interval (and not only its pseudo- inverse) 
and we have 

Q*oa^^(e) = F(e), Ve G [(/.q(O), eo]. (2.19) 



Identity (2.19) is still valid for eo < e < 0. Indeed, in this case, we have -F(e) = 0, 



and acpgie) > a^^Jeo) = meas(Supp(Q)), which implies that Q* o a(f)Q{e) = 0. The 
first identity of (2.17) is then proved. 

Q is a straightforward consequence of the first identity 



Now, the identity Q 
h 



then F 



of (2.17). Indeed, we first observe that 2 

= OandQ*'f'Q{x,v) ■ 



+ 



x) > <^q(0). If 4 



+ (Pq{x) > 



+ 



from the definitions of F and Q*'^ 



Note that this is essentially a characterization of spherical models 
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I |2 

If -^ — h 4'q{x) < 0, then we apply the first identity to e 
desired equality. The proof of Lemma |2.6| is complete. 



\v\' 



+ 



x) and get the 

D 



2.4. Monotonicity of the Hamiltonian under symmetric rearrangement. 

We are now in position to derive the monotonicity of the Hamiltonian under the 
generalized rearrangement which is the first key to our analysis and was already 
observed in the physics litterature, see pTj and references therein. Given / G f \{0}, 



by Lemma 2.1 we have (j)f ^ X and we will note to ease notation: 
Given (/) £ A', we define the functional 



1, 



which is well defined from Proposition |2.4| We claim: 



|2 
Il2 



(2.20) 



f2.211 



Proposition 2.7 (Monotonicity of th e Hamiltonian under the f*'^f rearrangement). 
Let / G iS \ {0} and f given by (2.20), then: 



nf)>jf,{<Pf)>nf). 

Moreover, %{f) = 'H{f) if and only if f = f- 
Proof. First compute for all {f,g) G £'■ 

nf) = U \v\'f-U wHi' 



(2.22) 



Kf 

aA2 



ng) + l\N<Pf - v</.,|li2 + ^^ (^ + hi^)) (/ - 9). (2.23) 



Replacing g hy f = f*'PS yields from (2.21): 



nf) = jf'ih) + Xe (^ + "^^^^0 ^^ ~ ■^*'^'^ '^'''^''' 



and hence (2.7) follows from 



+ <t>f{x)\{f-f)dxdv>0, 



(2.24) 



(2.25) 



with equality if and only if / = /. The proof of (2.25 



is reminiscent from the 



standard inequality for symmetric rearrangement, see |30] : 

/ \x\f* < f \x\f. 

JRG jRf> 

Indeed, use the layer cake representation 



f{x,v) 



il/ili- 

t=0 



h<f(x,v)dt 
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and Fubini to derive: 
■/r6 V 2 J 



+ H]if-f) dxdv 



II/IIl- 



t=o 



t=o 



t=o 



dt 



dt 



dt 



h<f(x,v) - ^t<f{x,v) 



+ (f>f j dxdv 



^f{x,v)<t<f{x,v) ^f(x,v)<t<f{x,v) 



+ (j)f j dxdv 



Slit) 



+ (/)j j dxdv 



S2(t) 



+ (t>f dxdv (2.26) 



with 



Si{t) = {fix,v)<t<fix,v)}, 
Observe from / G Eq(/) that: 

for a.e. t > 0, nieas(5i(t)) = meas(S'2(t)) 

We thus conclude from ^J^ and (|237J): Vt G (0, ||/||loo), 



S2{t) 



+ 4>f{x) j dxdv < meas(52(t))(/*oa^^) (t) 



S2it) = {fix,v)<t<f{x,v)}. 

(2.27) 

(/*oa0^)"-^(i)(ix(it;. 



•Si(t) 



Injecting this into (2.26) together with (2.15) yields: 

|2 



\V\ 



+ 0/ (/ - /) dxdv 



> 



l/IL 



dt 







Slit) 



+ (?!)/(x) - (/* o a^^y^it) ] dxdf > 



and (2.25) is proved. We also have the analogous inequality for S2{t): 
+ ^f]{f-f)dxdv> 



\v\ 

T 



/■||/IIl°c r 

/ ^*/ 

^0 Js 



dt I [{roa^y-\t) 

S2it) 



(t>f{x) I dxdv > 0. 



Let us now study the case of equality in (2.25). If 

,|2 



l- + cl)f{x)\{f-f)dxdv = 0, 

the above chain of equalities implies that for a.e t > 0, either meas(5'i(t)) = 
meas(S'2(t)) = or, a.e. (xi,wi) G Si{t), a.e (x2,t^2) G 5'2(t), 

^ + M^i) = if* ° HfrHt) = ^ + '^/(^2). 

The last assertion contradicts the fact that f(xi,vi) <t< /(x2, ^2)- Therefore, for 
a.e t G (0, ll/llioo), we have meas(S'i(t)) = meas(52(i)) = 0. On the other hand, 
||/||l°° = ||/*||l°° and hence meas(S'i(t)) = meas(52(t)) = for t > ||/||l°°- Hence 
meas(5'i(t)) = meas(5'2(t)) = for a.e. t > 0, which implies f = f- This concludes 
the proof of Proposition |2.7| D 
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3. Study of the reduced functional J 

In this section, we focus onto the functional on X: 

We claim that locally near ^q, J{<^') — J{^q) is equivalent to the distance of (/> to 
the manifold of translated Poisson fields (boi- + x), x S M^. 



Proposition 3.1 (Coercive behavior of ^ near (^q). There exist universal constants 
C0)<^0 > and a continuous map (j) ^ z^p from {H^, \\ • \\fji) -^ ^^ such that the 
following holds true. Let 4> ^ X with 

inf (110 - cpq{- - z)\\l^ + ||V<A - V</.q(- - z)\\l2) < 6q, (3.2) 

then: 

J{^) - J{<Pq) > co||V0 - V,/.q(- - z^)\\l2. (3.3) 



This section will be devoted to the proof of Proposition 3.1 which relies first on 
the second order Taylor expansion of J' at (pq, Proposition 3.3 and then on the 
coercivity of the Hessian which is the second main key to our analysis, Proposition 
3.6[ and corresponds to a generalized Antonov's coercivity property. 



3.1. Differentiability of J. Our aim in this section is to prove the differentiability 
of J at 4>Q and to compute the first two derivatives. 

Let us start with differentiability properties of the function (/> i— t- a^ defined in 
Lemma [2. 2 [ see Appendix [A] for the proof. 



Lemma 3.2 (Continuity and differentiability properties of (/) i— )• a,^). Let (f),(j) £ X 

and let h = <j) — (p. Then the following holds. 

(i) The function (A, e) i— )• a^j^\h{e) is a C^ function on [0, 1] x R^ and we have 

—a^+Xh{e) = -47r\/2 / (e - 0(x) - Xh{x)f_l^h{x)dx. (3.4) 

CA Jk3 

(ii) Let s G M^. Then the function A i— )• CL'.,y^y^{s) is differentiable on [0, 1] and we 
have 



^«AJ^) = ^^^7 • (3.5) 



"(jy+Xh 

We are now in position to differentiate the functional J . 



Proposition 3.3 (Differentiability of J). The functional J defined by (3.1) on X 

satisfies the following properties. 

(i) Differentiability of J . Let (p,(j) £ X, then the function 

A^J(,/. + A(0 -(/.)) 

is twice differentiable on [0, 1] . 

(ii) Taylor expansion of J' near (/)q. There holds the Taylor expansion near (/)q: 

ycpGX, 
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where 

r]{6) -^ as 6 ^ 0. 
Moreover, the second derivative of J at (pq in the direction h is given by 

D^J{4>Q){h,h) (3.7) 



\Vh\^dx- 
where Ilh is the projector: 

Ilh{x,v) = 



i^'(^ + 0Q(x; 



{h{x) — Ilh{x , v)) dxdv , 



1/2 



+ 4>q{x) - (t>Q{y) ) h{y)dy 

172 
+ <I)q{x) - (pqiy) ] dy 



(3.8) 



Remark 3.4. The projector lih given by (3.8) should be understood as the pro- 



jector onto the functions which depend only on the microscopic energy e(x, v) 
Proof. Let us decompose J into a kinetic and a potential part: 



1, 



J{c^)=jQ*{<t>)=nQ*n + l^\W<i^ 



? = \l W<t^?dx + M<t^) (3.9) 



with 



M<l>) 



2 

Note from Proposition 2.4 that Q**^ G 8 and is supported in ^-^ — h 1?^ < 0, thus 



(3.10) 



Let (j),(j) ^ X and let h 
Joi(j) + Xh). 



+ (l){x)jQ*'^ix,v)dxdv. 

j_;wi ljv^u. Ill 

-00 < Jo{^) < 0. 
We shall differentiate with respect to A the function 



Step 1. First derivative of Jq. 
Introduce the following primitive of Q*: 

G{s)= r Q*{a)da, (3.11) 

which is a uniformly bounded C^ function with bounded derivative, since by as- 
sumption Q (t hus Q *) is continuous and compactly supported. We first transform 



the expression (3.10 



of J'q. By making the change of variable in velocity e 



\vl 



+( 



and using (2.5), we get 

v^o(^) = / eQ* {a^{e)) a^{e) de 



uiincf) 



e{G o a^) (e) de 



[eG{a^{e))]l 



G{a^{e))de 



G{a^{e))de. 



Note that the boundary term is dropped thanks to the definition (3.11) and the 
following properties: 

r+00 
a^{mm(p) = 0, lim 0^(6) = -|-oo, / Q* (a) da = \\Q\\li < +00. 
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In order to differentiate i7o(0 + ^^h) witli respect to A, we now use (3.4) and tlie 
C^ smootliness of G to derive: Ve < 



G{a^+Xh{e)) = -A7rV2Q* {a^+xh{e)) / (e - (/<(x) - Xh{x))'/\{x)dx . 



d_ 
dX 

Recall that we have Supp (Q*) = [0, Lq], with 

Lq = meas (Supp Q) < +oo. 



Hence, from Lemma 3.2 (i), we deduce that there exists ei < 62 < such that 

{(A,e) G [0,1] X Ml : a^+xhie) G Supp(Q*)} C [0,1] x [61,62]. (3.12) 

Moreover, we have the following uniform bound: for all (A, e), 
d 



dX 



G{a^+Xh{e)) 



< 4:TrV2\\Q*\\Loo f (62 - (1 - X)(p{x) - X(f){x))^_/'^h{x)dx 

< 4^\/2||Q*||loo f (62 - (pix) - 4>ix)))^^h{x)dx < +00. 



Therefore, Lebesgue's derivation theorem ensures: 



d_ 
dX 



Jo{(t> + Xh) 



47r\/2 



d_ 
dX 



G(a, 



(p+Xh 



[e)) de 



1/2, 



Q* {a<t>+\h{e)) {e - 4>{x) - Xh{x)) jI_ h{x) dxde. (3.13) 



Step 2. Second derivative of Jq. 

Let us now compute the second derivative of j7o((/> + Xh) with respect to A. First, 



an integration by parts of (3.13) with respect to the variable e gives 
87r^/2 '■o 



d 



Jo{(t>+^h) 



5A^"^" ' 3 

Applying the change of variable s = a0+A/i(e), we obtain 



(5*'(a0+A/i(e))a0+A/i(e)(6-(/)(x)-A/i(a;))Y h{x) dxde 



d_ 
dX 



M^ + xh) 



8ttV2 



8ttV2 rLo 



[ 'dsQ*'{s) [ {all^^{s)-<l){x)-Xh{x)fl\{x)dx 
Jo Jrs ^ 

Q*'{s)g{X,x,s)h{x)dsdx, (3.14) 



with 



g{X, x, s) = (a^lxhi^) " '^(2^) " >^Hx))+'^- 
Recall that, by (3.12), the quantity e = 0^1 a/i('^) ^^^ ^^ restricted to some interval 



[61,62] in th is integral, with ei < 62 < 0. Moreover, as in Step 1 of the proof of 
one deduces from the decay at the infinity of </> and (j) that the domain 

{x eM^ : (t>{x) + Xh{x)} <e2 



Lemma 



3.2 



is bounded independently of A. Therefore, the variable x in the integral (3.14) can 
be restricted to a bounded domain. 
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Let us differentiate (|3.14) witli respect to A. From (3.5), one gets 



d_ 
dX 



giX,x,s) 



'2^"',f,l\h(-'^) - ^(^) - A/i(x))y^/i(x) 



+ i^i%lxhi^)-H^)-^hi^))+ 



1/2 Jk3 ^^ 



witli tlie uniform estimate: for all s £ [0, Lq], X £ [0, 1], x S M^ 



d_ 
dX 



g{X,x,s) 



~ 1/2 

< 3(e2 + |min(/)| + I min(/)|) / ||/i||L°o 



(3.15) 



Since the function s i— )• Q*{s) is monotone decreasing from ||Q||l«= to 0, the function 
Q*' belongs to L^{0, Lq), and hence the uniform domination (3.15 ) allows us to apply 
Lebesgue's derivation theorem and get: 



J^ Jo(0 + A/i) = 4ttV2 f " ds Q*'is) 



l«0+Ahl^J 



(t){x) - Xh{x)f/^{h{x)fdx 



-4^^/2 [ ° dsQ*'{s) 
Jo 



i«0+Ahi^J 



1/2; 



(x) — Xh{x))_^ h{x) dx 



{'^4,l\his)-4>{x)-Xh{x))l dx 



(3.16) 



Step 3. Identification of the first and second derivative of J at (J)q. 
Let (j) £ X and h = (p — (J)q . We claim that 

DJ{(t>Q){h) = 0. (3.17) 



Indeed, first remark from (3.9) that 



DJ{<i)Q){h) = DJo{(l)Q){h) + [ VcPq- Vhdx. 

Jr3 

Next, by ^J^ and (|2]T7j): 

DJo{cl)Q){h) = 47r^/2 /" f Q*{a^Q{e)){e-cl)Q{x))^fh{x)dxde 



(3.18) 



AttV2 



F{e){e — (f>Q{x))j^ h{x) dxde. 



Applying the change of variable e i— )• u = Y^2(e — <J)q{x)), it comes 



+ 00 



DjQ{cpQ){h) = Att I / 3 ^ (y + '^q(^)) KxWdudx 
Q{x, v)h{x) dxdv, 



where we used the expression (1.14) oi Q. Hence, from the Poisson equation, we 



deduce after an integration by parts that 

DJo{(l)Q)ih) = - I V^Q-Vhdx, 

Jr3 



which together with (3.18) implies (3.17). 
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Let US now identify the right second derivative of J at ^q. We have 

D^J{^Q){h,h) = D^Jo{qyQ){h,h)+ [ \Vh\^dx (3.19) 



and, by (3.16), 



Lo 



D^M4>Q){h,h) = AnV2 dsQ*'{s) / {a-^l{s)-<t>Q{x))X\h{x)fdx 



l-Lo y, {0'4,l{s)-4>Q{x)tl'^h{x)dx 

AttV2 dsQ*'{s)^^^ 

^° / {a^'is) - c^Qix))]/' dx 

Jm.3 "^y 



Using first the change of variable s >-^ e = a^ (s), (2.5) and F = Q 



we get 



D^Joi(f>Q){h,h) = 4ttV2 [ deF'{e) j {e-(PQ{x)f_l^{h{x)fdx 



-47r\/2 / F\e) 

J —oo 



(e- 4)q{x))^ h{x)dx 



x))_)_ dx 



We next apply the change of variable e i— )• u = \/2{e — (J)q{x)) to get 

D^M(t>Q){h,h) = I F'{e){h{x)fdxdv- I F'{e)h{x)Uh{e)dxdv 

JR6 JIR6 

\F'{e)\{h{x)-Uh{e)fdxdv, 



I |2 

where we used the shorthand notation e = '-^ — \- (J)q{x) and the fact that 11 given 



by (3.8) is the projector onto the functions which depend only on e. This together 



with (3.19) concludes the proof of (3.7). 



Step 4- Proof of the Taylor expansion (3.6). 

Let (f) € X and h = (f) — (pg. We first deduce from the fact that J{(J)q + A/i) is twice 
differentiable with respect to A that 

J(0Q + h)- J{<t>Q) = / (1 - ^)§y^J{<t>Q + A/l) d\ 



and hence: 



J(0Q + h)- J{^q) - -D''j{<PQ){h, h) 



(1 - A) {D^J{(t)Q + A/i) - D^J{^q)) {h, h) d\ 

= \\Vh\\l, / (1 - A) {D^Mcl^Q + Xh) - D^Jo{ct>Q)) 
Jo 

We now claim the following continuity property: 



h 



h 



||V/l|U2'||V/i|U2 



sup sup 



[D^McpQ + A(0 - cPq)) - D^McPq)) (h, h) 



(3.20) 



d\. 



(3.21) 
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as 



2,00 — )• 0. Assume (3.21), then 



lim / (1 - A) {D^M(I)Q + \h) - D^Jo{cI)q)) 



h 



h 



\Vh\\L2' \\Vh\\ 



L2 



dX = 



and ( 3.20[) now yields the Taylor expansion (3.6). 

Proof of (3.21 ). We argue by contradiction and assume that there exists e > 0, Hn, 



hn and A^ G [0, 1] such that 



\H.. 



71 L°° 



< 



n 



\VhJ 



L2 



and 



D^M^Q + XnHn){hn, K) " D^ Jo{(t)Q){K, K 



> e. 



(3.22) 
(3.23) 



We denote hn = XnHn- Recall from (3.16): 
D'^Jq{4>Q + K){hn, hn) = 



A^:^/2 / dsQ*'{s) 
Jo 



^"'<t>Q+h„y^) 



+ hn){x))/ {hn{x))'^ dx 



1/21 



47r\/2 / dsQ*'{s)^ ^ 

I J 



(%+h„('5) - i(t>Q + hn){x))^ hn{x) dx 



("</'Q+/in(''^) ~ ^^Q + ^"(^))+ ^ ^^ 



Let us analyze the sequence e^ = a, , ^ (s). We claim that: 



VsG(0,Lo), lim en = aMs). 



(3.24) 
(3.25) 



Indeed, we observe 



87r^/2 
+/i„(e„) > -^7— I (e„ 



a^) - \\hn\\L^)+ dx > 



n 



which yields e„ < a, (s) + 1/n. Similarly, we also have e^ > a^, (s) — 1/n and 



(3.25) follows. 



Let us now pass to the limit in (3.24). Note first that the domain of integration in 



X of these integrals is uniformly bounded as n — )• +00. Indeed, the set of integration 



IS 



Dn{en) ■■= {xeM.^ : 4>q{x) + hn{x) < en} C {x G M^ : ^q{x) < e„ + 1/n} , 

which is bounded for n large enough, since e„ < oT (Lq) + 1/n < 20,7 (Lq), and 
the continuous function (J)q converges to zero at infinity. 

Now the local compactness of the Sobolev embedding H^ ^->- Lf^^ for 1 < p < 6 
implies that there exists h G H^ad such that -up to a subsequence- 



hn ^ h in Ljq^ as re — )• +(X). 



(3.26) 



20 



M. LEMOU, F. MEHATS, AND P. RAPHAEL 



Hence, for all s G (0, Lq) and for z = 0, 1, 2, ( |3.25 ), (3.26) ensure: 

(%l+hj^) - (^Q + K){x)flXK(x)y dx 

(a7 (s) - 0q(x))Y {h{x)y dx as n -^ +oo. 
Moreover, by Caucliy-Scliwarz and aT , ^ (s) < 0: 



kl<-R 



(a<A^ /,„(s) - 0Q - /in)+ /in dx 



1/21 



(a,in+hJs)-'/'-^")+^^^ 



^ IKl+hS^) 



^1/2, 



""-/■Q+Zln^ 



< 



|z|<_R 



LOO + ||/i„||L°°)^''^(/in)^da; < 1. 



Recall now that the function Q*' is L^ on [0, Lq], since Q* is decreasing and bounded. 
Therefore, Lebesgue's convergence theorem applied to (3.24) yields: 

A similar argument gives: 

D^Jo{cI)q){KX) -^ D^Jo{cl)Q)(h,h) (3.27) 



as n — ?■ +00. This contradicts (3.23) and concludes the proof of (3.21) 
The proof of Proposition 3.3 



is complete. 



n 



Remark 3.5. We have proved in this last Step 4 that for all sequence hn bounded 
in H^, after extraction of a subsequence, we have the strong convergence (3.27). 
Hence the quadratic form D'^J'q{4'q) is compact on H^. 

3.2. A new Antonov type inequality. We now turn to the second key of our 
analysis which is a generalization of the celebrated Antonov's stability property -see 
Proposition 4.1 in [29j for a precise statement-: 

Proposition 3.6 (Generalized Antonov's stability property). Let Q satisfy the as- 
sumptions of Theorem\1.2\ and consider the linear operator generated by the Hessian 



(3.7).- 



Ch = -Ah- \F'{e)\{h-Uh)dv. 

Then C is a compact perturbation of the Laplacian operator on H^ and is positive: 
yheH^, iCh,h) = D^J{(l)Q){h,h)>0. (3.28) 

Moreover, 

Ker{C) = {h£ H^ with Ch = 0} = Span{dxi(t)Q)i<i<z- 
In particular, there exists cq > such that 

2 

(3.29) 



'ih(^H\ {Ch,h)>co\\Vh\\l,--y([ hA{d, 
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Remark 3.7. The fact that the kernel is completely explicit and purely generated 
by the symmetry group is remarkable and reminiscent from similar statements in 
dispersive equations, see Weinstein |44j . the coercivity on the radial sector being 
always the most delicate problem. 

Proof of Proposition \3.(^ 

Step 1. Positivity away from radial modes. 

Let h £ H^ad^ ^"^^ ^^^ ^^ introduce the projection of h onto the radial sector 

where S^ denotes the unit sphere in M^ and da denotes the surface measure on S^ 
induced by the Lebesgue measure. We have the decomposition 

h = ho + hi, hoe Hl^d, hi e {Hl^d)^. 



The angular integration in (3.8) ensures Yihi = and thus 



hi 



{Ch,h) = {ChoM + I |V/ii|2- / Vq. 
VQ{r)= f \F'{e)\dv = iTrV2 [ \F'{e)\{e - (l)Q{r))]/^ de, 

JR3 Jcho(O) 



where we applied the change of variable e = '-^ — \- (f)Q{r). Since F\e) < and F 
is bounded on [(/>q(0),0], the function \F'\ belongs to L^. Therefore by dominated 
convergence, the function Vq is continuous. Moreover, since F{e) = for e > cq 
and since (pq is strictly increasing, we have: 

Supp(FQ) = [0,(<AQ)-neo)]. 
Hence, Vq being continuous and compactly supported, the Schrodinger operator 
—A — Vq is a compact perturbation of the Laplacian on H^. Observe that (/)q (and 
also dxi4>Q for i = 1, • • • ,3) belongs to i/^(M^). Translating the 0q equation yields: 

A(J)q{x + xo) = Pq{x + xq) = -^— / \F'{e)\{e - (Pq{x + xo))+ de 

■J J-oo 

and differentiating this relation with respect to xq yields at xq = 0: 

£(V</.q) = 0. (3.30) 

We now claim from standard argument that this implies the positivity of C away 
from radial modes, see ^44| for related statements: 

v/^e [n^adY , (£/i,/i)>o, (3.31) 

and 

{h G (H^ad)^ with Ch = 0} = Span(c>,,</.Q)i<i<3, (3.32) 

Let us briefly recall the argument. Let us decompose h £ (H^ad) ^^^° spherical 
harmonics, 

h = y^^y^^hkjYkjjx) 

k>l j 
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where x = - is the spherical variable and —Ag2Ykj = XkYkj. Then the radiality of 
Vq ensures the orthogonal decomposition 

£/i = ^ ^ Akhkj 
fc>i j 

with 

^-VQ{r), Xk = k{k + l). (3.33) 



A, 



-dl - -dr 



For A; = 1, we have V(/)q = 0Q(r)a; and (3.30) implies AicpQ = 0. Since (J)'q > for 
r > 0, (pQ G H^, <J)'q is from standard Sturm Liouville results the ground state of 
Ai which is thus positive with kernel on H^^^ spanned by (P'q. Now (3.33) ensures 
that Ak is definite positive on H^^^ for k>2 and (3.31), (3.32) follow. 



Step 2. Coercivity away from radial modes. 
We now claim: 



3 9 

\fhe{Hl,)^, {/:h,h)>ci\\Vh\\l,--Y,( [ hA{d,^<t>Q)) (3.34) 

for some universal constant ci > 0. Let us briefly recall the argument which is 



standard. From (3.31), 



0^ > 0. 



/ = inf|(£/i,/i), heiH^^a)^, [ VQh^ = l, [ hA{d,^ 

We argue by contradiction and assume / = 0, then there exists a sequence /i„ E 
(HU^ with 

f VQhl = l, f \Vhn\^- f VQhl<^, f KAid,^cPQ) = 0. 

From Sobolev embeddings, hn ^ h in L^^^, 1 < p < 6, up to a subsequence. 



Moreover, from (3.30), A{dxi4>Q) is compactly supported and in L from which 



passing to the limit yields 

{Ch,h) < 0, 



/iA(a,, 



0, 



Vgh' 



(3.35) 



and hence h ^ attains the infimum. From Lagrange multipier theory, we thus can 
find (Aj)i<j<3 with 

3 



i=l 



Taking the inner product with h yields Aq = 0, then with dx^cpQ yields Aj = 0, 
and thus Ch = 0. From (3.32), h £ Span{dxi4>Q)i<i<3, but this contradicts the 
orthogonality relations (3.35), and (3.34[) follows. 



Step 3. Strategy: Hormander's proof of Poincare inequality. 

The relative compactness of C with respect to A in H^ follows from Remark 



3.5 



It thus remains to prove (3.29) which from (3.34) and the Fredholm alternative is 
equivalent to: 

Wh e H^^a, /i / 0, {Ch, h) > 0. (3.36) 
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Our main observation is now from (3.7) that (3.36) is nothing but a Poincare in- 
equality with sharp constant, and we now claim that we can adapt the celebrated 
proof by Hormander [20', '21] to our setting. Hormander's approach involves two key 
steps: the introduction of a self-adjoint operator adapted to the projection involved, 
and a suitable convexity property. The operator will be given by 

Tf(e,r) = , ^ drf (3.37) 

'^ ' rV2(e-</>Q(r)) ^ ' 

which essentially satisfies the requirement 

Uh = implies h G Im(T), 

and the convexity will correspond to the lower bound: 

with 

g{r,e) = \r^j2{e-(l)Q{r))\ . 

Note that the original proof of Antonov's stability criterion can be revisited as well 
using the transport operator t = v- V^. — ^x'pQ " V^, in the radial case as differential 
operator and whose image can be realized in the radial setting as the kernel of the 
full projection including the kinetic momentum i, see |19] . |29) for more details. 



Step 4- Integration by parts. 

Recalling that (j)Q{r) is strictly increasing, for all e G (0q(O),O), we shall denote 

Let h G H^ad ^°^ zero. Let 

U = {(r,e): r > 0, e G (</.q(0),0), e - (/.Q(r) > 0} 
= {(r,e): eG(0Q(O),O), rG(0,r(e))}. 
Given e > 0, we let < XeC^) < 1 be a smooth cut off function such that 
Supp(xe) C ((/>Q(0) + e,eo-e), Xe = 1 on [(J)q{0) + 2e,eQ - 2e]. 



We let 
Observe that 



re = r{(j)QiO) + e). 



r{e) 



/O 
and hence the radial interpolation estimate 



e - <f>Q{T) r'^dr > c^ > on Supp(xe), 

eolation estimate 

||Vr/i(r)|Uoo(R3) < ||V/i||i2(K3) (3.39) 

and (3.8) ensure: 

\Uhie)\ < Ce on Supp(xe). (3.40) 

Let us then define onU =U r\ (0,r(eo)) x (0q(O), cq): 

/(r,e)= f {h{T) - n/i(e))^2(e - (pq{t)) r^dr. (3.41) 
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Then / is C^ with respect to the variable r > with 

Tf = h-Uh 



(3.42) 



on U, where T is given by (3.37). Moreover, (3.39) and (3.40) yield the bound at 
the origin: Ve G Supp(xe), 

|/(r,e)|<Qr5/2, (3.43) 



and from (3.8) we get 



+ 00 



/ (r(e), e) = V2 {h{T) - Uh{e)){e - ^q{t))'/^ r'dr = 0. 



Hence, near the boundary r = r(e), we estimate using (3.40): Vr > r^, Ve G 
Suppxe, 



\f{r,e)\ 



r{e) 



(hir) - Tlh{e))^2{e-<PQ{T)ydT 



<Ce{e-(t^Q{r)f/\ (3.44) 



where we used e — 4>Q{r) ~ C{r{e) — r) deduced from (p'nir) > i?^Q('^e) > 0. 

We now integrate by parts from (3.42) using the cancellations at the boundary 



of W given by (3.43), (3.44) and the bounds (3.40), (3.39): 



/ Xs\F'{e)\{h-Uhfdxdv 

JR6 



Xe\F'{e)\de {h-Uh)TfJ2{e-cl)Q{r)r^dr 

;o) Jo ^ 

/•O /■''(e) 

levrM Xe\F'{e)\de j {h-Uh)drfdr 
J<t>Qio) Jo 

-levr^ fxe\F'ie)\fdrhdedr. 
Ju 



We now use Cauchy-Schwarz together with the identity 

PQir) = ^ /° |i^'(e)|(e - <PQ{r))fde 



0q(o) 



to estimate: 



Xs\F'{e)\{h-Uhfdxdv 

< {4-Kf/'^\\Vh\\L2 

< (4vr)3/2||V/ib2(R3) 







-^ ( / Xs\F'{e)\fde 



1/2 



(pgir) 



8ttV2jo 
I|V/i||l2(k3) I 3 / XePQ{r 



Jo r^ Jd>n(r) 



f 



1/2 



't>Q{i') 



f 



, =- \F\e)\dxdv\ 



"(e-0Q(r))3/2 

1/2 



de 



(3.45) 
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We now claim the following Hardy type control: 



^ Xe\ PQ + 



r J r^{^2{e-ct>Q)y 



\F'{e)\dxdv < I Xe\F'{e)\{Tffdxdv. 



Assume (3.46), then (3.42) and (3.45) yield: 



Xe\F'{e)\{h-Yihfdxdv+?> [xe- 



f 



r r^{y^2{e-^Q)Y- 



(3.46) 
\F'{e)\dxdv < I \Vh\^dx. 



Letting e — )• now yields {Ch, h) > 0. Moreover (Ch, h) = implies / = in 
U, thus h{r) = n/i(e) on U and {Ch,h) = J |V/ip = and thus h is zero. This 
concludes the proof of (|3.36 ) . 



Step 5. Hardy type control. 



The Hardy control (3.46) is a consequence of the convexity estimate (3.38). Indeed 



let g he a given smooth function in U, let f = qg and compute: 

{Tff = {gTq + qTgf=g\Tqf+g{Tg)T{q^) + q\Tgf 

= g\Tqf + T{q^g{Tg)) - q\{Tgf + gT^g) + q^Tgf 
-.'2„rr\ „2„rr2„ rr(„2„rr„\ ^ g 2 



> T{q'gTg) - q'gT'g = T{q'gTg) 



9 



r. 



We now look for g such that 



> 



9 



lrVW^M]'r^ ^ 



(3.47) 



(3.48) 



Let 



and thus: 



T^g 



2(e 



so that Tg{r,u) = -^ 



r^u r'^v? 



dug, 



r^v? 



dl^g dr-g 



PQ d q + ^ 
'Jug T^ „/i„.4 



r'^ifi 



drg + 4.- dug- 2udj,ug 
r 



k' \2 



+ 



J-iyp 



[udlu9 - 2<9„g] , 



where we used the Poisson equation satisfied by </>q. The choice g = r'^u^ yields: 

T'g 3 ' 



g r^u^ \ T 



Injecting this into (3.47) and integrating on hi yields: 



J Xe\F'ie)\iTffdxdv > 3Jxe(pQ + ^] 



f 



r\^2{e-<PQ)f 
+ Jxe\F'{e)\T(^f^^dxdv. 



\F\e)\dxdv 
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The bounds (3.43), (3.44) now justify the integration by parts 

[xe\F'{e)\T (f'^) dxdv = IGvr^ T Xe\F\e)\de f ' dr ( f^) dr = 
J \ 9 J J<t>Q(o) Jo \ 9 J 



and (3.46) follows. This concludes the proof of Proposition 3.6 



D 



3.3. Proof of Proposition 3.1 We are now in position to conclude the proof of 
Proposition |3.1| which is a classical consequence of modulation theory coupled with 



the coercivity estimate (3.29). 



Step 1. Implicit function theorem 

Given a > 0, let [/^ = {</, g H^{m?); ||V0 - V0Q||i2 < q}, and for (p G H^, 
z G M^, define 

e,{x) = <j){x + z)-<PQ{x). (3.49) 

We claim that there exists a > 0, a neighbourhood V of the origin in M^ and a 
unique C^ map Ua — )• V such that if G [/„, there is a unique z G V such that e^ 



defined as in (3.49) satisfies 

Vl<i<3, / e,A{d^^(f)Q)dx = 0. 

Moreover, there exists a constant C > such that if u G Ua, then 

\z\ + \\Ve,\\L2 <C\\V(P-V^q\\l2. 
Indeed, we define the following functionals of (i;^, z): 

Ti{(p,z)= ezA{d:,^(j)Q)dx, l<i<3 

Jr« 



(3.50) 
(3.51) 



and obtain at the point {(p, z) = {(pq, 0) 



dzj 



6ij\\Vdx 



2 
L2- 



The Jacobian of the above functional is — n?^^||V02;.(/)Q||^2 < 0, hence the implicit 
function theorem ensures the existence of a > 0, a neighbourhood V of the origin 
in M^ and a unique C^ map Ua -^ V such that (3.50) holds. 



Step 2. Conclusion 
Let (/> G Af with 

inf iU - <Pq{- - z)U^ + \\V<P - V(/>q(- - z)\\l.) < 5o 

for some small enough 5o > to be chosen later. Then there exists zi such that 



- zi)\\loo + ||V</. - V0q(- - zi)\\l2 < 26o. (3.52) 

For 6o < ^ small enough, we may apply Step 1 to (j){x + zi) and find Z2 G M'^, 



e G H^ satisfiying the orthogonality conditions (3.50) and the smallness (3.51) such 
that (j)(x + zi) = 



+ e){x — Z2), or equivalently 

(f){x) = {(j)Q+£){x- Z^), Z^ = Zi+Z2. 



(3.53) 



In fact, for 5o small enough, a shift z^h satisfying (3.53), the orthogonality conditions 



(3.50) and the smallness condition (3.51), is unique. This is a simple consequence 
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of the uniqueness of the pair (z2, £23) i^^ Step 1. The continuity of the map </)—)• z^ 
from {H^, II • 11^1) - 



e i° 



< 
< 
< 



> M^ then follows. Moreover, from (3.51), (3.52): 

x + zi + Z2) -</'q(x)||loo 

x + zi + Z2)- (pqix + 2:2)||loo + \\(I)q{x + Z2) - (/)q(x)||loo 

x + zi) - 0q(x)||loo + C|2;2| 

x + zi)- (J)q{x)\\l^ + C\\V(t){x + zi) - V(I)q{x)\\l2 < CSq. 



Provided 6q small enough, we may now apply the Taylor expansion (3.6) together 



with the coercivity (3.29) and the orthogonality conditions (3.50), and obtain from 

Co, 



the translation invariance of J^: 

Ji<P)-Ji(f>Q) = Ji(f>Q+e)-J{^Q)>co\\Ve\\l2-r]{\\e\ 
Co, 



iVel 



L2 



> 



IVel 



L2 



> 



-||V0-V(/>q(--z^)|| 



2 
L2- 



This concludes the proof of Proposition 3.1 



4. Compactness of local minimizing sequences of the Hamiltonian 

The aim of this section is to prove the following compactness result which is the 
heart of the proof of Theorem |1.2[ 



3.1 



Propositio n 4 .1 (Compactness of local minimizing sequences). Let 60 > be as in 

Let (p ^ Zff, the continuous map from {H^, \\ ■ \\^i) -^ I^^ build in 
Let fn be a sequence of functions of £, bounded in L°°, such that 

inf (||<A/„ - <Aq(- - ^)||l- + ||V<A/„ - V<^q(- - z)\\l2) < 5o, (4.1) 



Proposition 
Proposition 



3.1 



and 



Then 



limsupnifn) < n{Q), /: ^ Q* m lH^+) 

71— >+00 



as n ^ +00. (4.2) 

(4.3) 



{l + \v\)\fn-Q{x - z^jJ\^Q as n -^ +00. 

Proof. Step 1: Compactness of the potential 

We first claim the following quantitative lower bound which generalizes the mono- 



tonicity formula (2.22): let f ^ £ such that (j)f satisfies (3.2), let z^, given by 
Proposition |3.1[ then 



L-lir - QIIlI > Co\\V(t>f - V<Aq(- - Z^j)\\ 



L2- 



(4.4) 



n{f)-n{Q) + 

Indeed, 

■H{f) -nQ)> Jf* i<t>f) - Ji^Q) = Jr i^Pf) - Ji4>f) + J{4>f) - J{<^q), (4.5) 
where we have used that T-L{Q) = J{4>q)- Now, we recall that 

|V(/>|2(ix., 



JA<P) 



^^ + Af*^{x,v)dxdv + ]^ 



and deduce from the change of variables formula (2.10) that 

/•+00 
Jf,{ct>f)-J{ct>f)= / a-]{s){f*{s)-Q*{s))ds. 
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Since \a, (s)| < — min cftj = \\4>f\\L°°, we have 

Jf*ih) - Jih) > -Whh-Wf* - Q*\\l^- 



Inserting this estimate into (4.5) and using Proposition (4.4) yields (4.4) . 



Let us now consider a sequence /„ G i5 satisfying the assumptions of Proposition 



4.1 then (4.4) apphed to /„ ensures: 

l|V'^/n(-+^0/J-V</'QllL2^O, 



as n — )• oo. 



(4.6) 



Step 2: Strong convergence of fn to Q 

To ease notations, we shall still denote by fn the translated function /„(. + z^, , v) 
We then observe the identity: 

nfn)-n{Q) + ^\\VcPu-v^Q\\l,= 

which implies, from (|4.2|) and (|4.6|), that 



+ <pQix) ] (fn - Q)dxdv (4.7) 



■/M6 V 



+ <Pq{x) I {fn — Q)dxdv — )• 0, as n — )• oo. 



(4.8) 



Now, we observe from the change of variables (2.10) that 



Since |a7Vs)| < -(/'q(O) we get 



x)] {Q - f*J'')dxdv 



<i</'Q(o)iiig*-/:i 



which implies from (4.2) that 



/ ( 2 "^'^'^'•^'') *^^~'^"'^'''*^^^^~^^' asn-^oo. 



(4.9) 



Summing (|4.8|) and (|4.9|) yields 

Tn-- 



+ (I>q{x) {fn - fn'^)dxdv ^0, as n ^ OO. (4.10) 



We now argue as in the proof of (2.25), and write ( |4.10 ) in the following equivalent 
form 



+ 00 



dt 



i=0 



S^{t) 



+ (t>Q{x) I dxdv 



s^{t) 



+ 4>q{x) I dxdv 



where 



^0, 
(4.11) 



S'^{t) = {{x,v) G R^f7''{x,v) < t < fn{x,v)}, 

S^{t) = {{X,V) G R',fn{x,v) < t < f:'^''{{x,v)}. 



From (2.15), we have 

|2 



\V\ 



+ Mx) > {rnOa^arHt), y{x,v) G S^t). 
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Thus 



T > 



+00 



dt 



{fn°HQ) ^{t)dxdv 



+ (I)q{x) dxdv . (4.12) 



it=o \Jsi^{t) Jssit) 

As a consequence of the equimeasurabihty of fn '^ and /„, we know that 

meas(5]'(t)) = meas(5J(t)), 



and then (4.12) gives: 



T > 



+00 
t=0 -'5J{t) 



(/:°a0Q)''W 



+ 



dxdv. 



(4.13) 



From (2.14), we have 



(/: ° «<^J-Hi) > ^ + </'q(3^), V(x, t;) G 52"(t) 



Thus, from ( |4.10[ ) and ( |4.13| ), we get 



A. 



(/>«<^g)-'(i) 



+ 00 2; 



as n — )• +00, for almost every {t,x,v) G 
now claim that this implies 

|2 



ls^^t)ix,v)^0 (4.14) 



X M'^ X R'^ (up to a subsequence). We 



Bri 



(Q*oa^^)-i(t)-^ + 0Q(x) 



%^{t)(^''^)^0' 



(4.15) 



X M^ X M^, where 



as n — )• +00, for almost every {t,x,v) G 

S2(t) = {{x,v) e M6,/„(x,i;) < t < Q{x,v)}. 
To prove ( 4.15[ ), we write 

s^ = {s^\s'',) u (52^ n 5^) , 5^ = {s:^\s^) u (^2" n 5^) , 

and get 



■^n -On 






(4.16) 



We shall now examine the behavior of each of these terms when n — )• oo. We first 
observe that for all g,h £ L^(R.^) with g > 0, h > 0, we have 

r+co r- 

I vneas {g < t < h} dt = / (/i — (j)-!.^^^^ < H^f — /i||^i, 
Jo ^R6 



(4.17) 



and thus from (4.2): 

f+OO 



meas(52"(t)\52 (t))<it < \\f7'' - Q\\l^ = Wf^ - Q 



<t>Q 



0. 







Using in addition the estimate 
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we deduce that the first two terms of the decomposition (4.16) go to almost 



everywhere when n goes to infinity. We now treat the third term and sliow tliat, 
for all (t, X, v), 

l™,Sf [(fn ° «0q)"'W - (Q* ° «<^g)-'W] ^SS(t)nS-At) ^ 0- (4.18) 



To prove (4.18), we first use the strong L^ convergence (4.2) to get 



Ve E i<PQ{0),0)\A, /:(a<^Je)) ^ Q*ia^Q{e)), (4.19) 

where ^ is a zero-measure set in M, and claim that the monotonicity of /* in e and 
the continuity of Q* in e ensure: 

Ve G (0q(O),O), f^ia^^ie)) ^ Q*(a^Je)). (4.20) 

Indeed, let e G ((/)q(0),0), and {xp,yp) G ((/>Q(0),0)\yl such that Xp < e < yp and 



e, yp ^ e. As f*o a^„ is decreasing, we have 



fn ° HqiVp) <fn° Hqi^) < fn ° Hqi^p)- 



From (4.19) we then get 



Q*{a^Q{yp)) < liminf/*oa0^(e) < lim sup /* o a^^ (e) < Q*{a^Q{xp)). 

n— !>oo n— >oo 

Now we pass to the limit p — )• c« and use the continuity of Q* o a^^ to get the claim 



(4.20) 



Now, we turn back to the proof of (4.18) and fix {t,x,v). Take then any e such 
that 

(/>Q(0)<e<0, and Q*{a^^ie)) > t, (4.21) 



which implies from (4.20): 



fn{a<PQ{e)) >t, 



for n large enough. Using the definition of the pseudo inverse given in Lemma 2.5 
we then obtain e < (/* o a0„)~^(t) for n large enough, and hence 

e<liminf(/*oa<^ )-^(t). 



Since this equality holds for all e satisfying (4.21 ), we conclude from the definition 
of the pseudo inverse (Q* o a^„)~^(t) that 

liniinf(/: o a^^rHt) > {Q* o a^^)-\t), 



which yields (4.18). 



We now turn to the decomposition (4.16) and get from (4.18) 



liminf(A„ — Bn) > 0, for almost all {t,x,v). 



Finally, observing that B„ > and using (4.14), we conclude that (4.15) holds true. 
Observe now that 

|2 

x)\ >t. 



t < Q{x,v) implies Q{x,v) = F ( ^ — h 



\v\^ 



By the assumptions of Theorem 1.2, e — )• F{e) is continuous and strictly decreasing 
with respect to e 



+ <Pq{x) for {x^v) G {Q > 0}, and thus: 



t<Q{x,v) implies (Q*oa0Q)^ '(t) 



x) >0. 
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We then deduce from (4.15) and from S2{t) = {{x,v) : fn{x,v) <t< Q{x,v)} that, 



up to a subsequence extraction, 

^{fn<t<Q} ^ 0; as n ^ oo, 
for almost every {t,x,v) G M^ x M^. Now from 1|j^<^<q| < lu^qy and 

/ / ^{t<Q}dxdvdt =\\Q\\ LI <+oo. 
Jo JR6 

we may apply the dominated convergence theorem to conclude: 



'^{fn<t<Q}dxdvdt — )• as n — )• oo. 



Injecting this into (4.17) yields 



/ {Q ~ fn)+dxdv — ;■ as n — )■ oo. 
Now we claim that, using /* — )• Q* in L^, this implies 

/ ifn — Q)+dxdv — )• as n — )• oo. 

JR6 
Indeed, we write 

ifn - Q) + dxdv </(/„- fj'') + dxdv + [ {fj'^ - Q) + dxdv 

JRS JR6 

{fn^''<t<u}dt+\\rJ'^-QU. 
{fn<t<fJ"'}dt+\\r^-Q*\\j^, 

[fj^ - f^)^dxdv+\\f:-Q*\\L. 

< [ {Q-fn) + dxdv+ [ {f*J<^-Q)^dxdv+\\f:-Q*\\Li 
JRS JR6 

< / {Q-fnhdxdv + 2\\f:-Q*\\Li 



r+oo 

< / meas 

r>+00 

meas 
/o 



where we repeatedly used (4.17) and the fact that fn'^€ Eq(/„) implies 
Vt > 0, meas {/:'^« <*</„}= meas {/„ < t < f^'^'' } . 



(4.22) 



(4.23) 



As /* —7- Q* in L^, we then conclude that (4.22) implies (4.23). Finally adding 
( 4.22D and ( |4.23[ ) gives 

ll/n - QIIli -^ as n -^ +00. 

Furthermore, (4.2) and the strong convergence ^4>f„ — ?■ V(/)q in L^ imply: 



bl^/n 



|t;| Q as n — )• +oo, 



Together with the a.e. convergence of /„, this yields the strong L^ convergence of 
I^P/n to l^pQ. Note that the uniqueness of the limit now implies the convergence 



of all the sequence /„ which completes the proof of (4.3). 



This concludes the proof of Proposition 4.1 
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D 



5. Non linear stability of Q 



We now turn to the proof of the nonhnear stabihty result stated in Theorem 1.2 



which is a direct consequence of Proposition |4.1| and the known regularity of weak 
solutions to the Vlasov-Poisson system. 



Proof of Theorem |1.2| . 

Step 1. Continuity claim for weak solutions 



Let fo ^ £ and let f(t) G iS be a corresponding weak solution to (1.1). By the 



properties of weak solutions of the Vlasov-Poisson system [3 E| j we have 

Vt>0, /(t)GEq(/o), nf{t))<'H{fo). (5.1) 

We claim: 

(j)f £C{[0,+oo),L°°{R^)nH\R^)). (5.2) 

Note that this implies from Proposition |3.1| that 

t — )• z^p M is continuous. (5-3) 



To prove (5.2), recall that / G C{[0,+oo),L^) (see [313) and hence (5.2) follows 
from: \/f,g e £, 

W^h - W,||l2 + Uf - Ml^ < Cf,g 11/ - sIliC, (5.4) 

where Cf^g only depends on ||/||£: and H^H^-. Let us prove (5.4). First, from Holder: 



VxG 



\(l)f - cl)g\ix) 



pfiy) - pgiy) 



dy 



,5/6 



\\pf- PgWi^rAPf- Pg 



47r|x — y 
and from Hardy-Littlewood-Sobolev: 

||V0/ - V^g\\L-i < \\pf - Pg||L6/5 < \\pf - Pgllil'IlP/ - Pg||^5/ 

Second, by interpolation. 



,1/6 



,7/12, 



5/12 

3- 



\\Pf - Pgh^/^ < 11/ - 9&\v\\f - g)\\f < Cf^g- 



Since \\pf — PgW^i < ||/ — S'Uli, this yields (5.4) and the continuity (5.2) of 
follows. 

Step 2: Conclusion. 



An equivalent reformulation of Proposition 4.1 is the following: for all e > small 



enough, there exists i] > such that if f £ £ with 

|ir-g*||Li<^, II/I|l-<IIQI|l-+m, n{f)<n{Q) + ^ (5.5) 

and 

inf (110^ - cPgi- - z)U^ + ||V<A/ - V0q(- - z)\\l2) < 5o, (5.6) 

then 

ll(l + H')(/-Q(--%.))llLi<e- (5.7) 

Let e > and let ry > be the associated constant. We consider an initial data 
/o G <? with 

II/o-QIIlK^, ||/o||l-<||Q||l-+M and n{fo)<'H{Q)+v 
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and a corresponding weak solution /(t) of (1.1). Observe that, by the contractivity 

(5.8) 



of the symmetric rearrangement in L^ (see |30j), we have 

\\fo-Q*\\L^ = \\fo-Q\\L^<V- 
Moreover, (|5.4|) implies that, for r/ small enough. 



\\VcPf,-Vcl>Q{--z.)\\L. + Uf{0) 



No ' 



\l^ < y- 



From (5.1), we first deduce that the corresponding solution f{t) of (1.1) satisfies 
(5.5 ) for all t > 0. Hence, if we prove that 



Vt>0, ||V0/(i)-V(/)Q(--z^^,(t))||i2 + ||0/(t)-(/.Q(--z^^.(t))||L- <6o, (5.9) 



then (5.7) holds true for all t > 0, which is nothing but (1.17). Now (5.9) follows for 



r/ > small enough from a straightforward bootstrap argument using the continuity 
(5.2), (5.3) and the bound (5.4). The proof of Theorem 1.2 is complete. D 



Appendix A. Proof of Lemma 3.2 



Proof. The proof is similar to the one in |29j and we briefly sketch the argument 
for the sake of completeness. Recall that the set X is convex, thus (p + Xh = 
(1 — A)0(x) + X(p{x) belongs to X for all A G [0, 1] and a^fij^xh is well-defined. 

Step 1. Proof of (i). 

Let ei < be fixed. For all e < ei, we consider the domain 

D^+Xh{e) = {x G M^ : (</. + \h){x) < e} . 



From (2.4), we have 



a^+Xh{e) 



8itV2 



4>{x) — Xh{x))_/ dx. 



We clearly have 

D^+Xhie) CD^{ei)\JD^{ei). 

Since (j){x) and (l){x) go to zero at the infinity, D(f,{eo) and D-r{eo) are bounded. 
Hence for all e < ei, D^^xh{e) is contained in a fixed compact domain of M^. As in 
addition the functions <j) and <j) are continuous, the Lebesgue dominated convergence 
theorem may thus be applied to obtain the continuity and the differentiability of 



(icp+Xhi^) with respect to A and e. The expression (3.4) follows. 

Step 2. Continuity of the function A i— )• o,^_i_xhi^)- 

Let s G M*^_. In this step, we prove that the function A i— >• o7,^^(s) is continuous. 
To this aim, we consider a sequence A^ G [0, 1] converging to Aq as n — )■ +00 and 
prove that aT^^ ^(s) converges to aT^^^ h(s)- We set 



a 



-1 



(s) G (min(</) + A„/i), 0) C (mine/) + mincj), 0). 



Hence, up to a subsequence, e^ converges to some e < as n — )■ +00. 
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Let us prove that e < by contradiction. Assume that e = 0. For n large enough 



such that ^ < An < 



a(j,+x„h{en) 



l+Ao 

2 ' 



we have 



> 



87r\/2 
3 



JlR3 ^ 



~ N 3/2 
(1 - A„)(/)(x) - Xn4>ix)j dx 

1 — A A ~ \ ^^"^ 
Y~^(t){^) - -^4>{x) \ dx = a^(e„), 



—2^(f){x) + -y(I)(x). From Lemma 



2.2 



we have Hmt-5>o- (^ihit) = +oo, 



where tlj{x) 

which imphes that lim„_j>+oo ^■^(en) = +oo, a contradiction. 

Therefore, we have e^ — ?• e < 0. The continuity of (A, e) i— )• a(/,+A/i(e) proved in 
Step 1 gives that 

s = a^+\„hien) -^ a^+Xohie) as n -> +oo. 

Thus e = oI_^_;^ hi^)- This ends the proof of (ii). 



Step 3. Differentiability of X \-^ a,. 



(t>+\h 



.S). 



Denoting (J)q = (p + X^h and (j)\ = (j) + Xh^ we write 



X 

where we have set 
Ai{X) 



Ai{X)A2{X), 



X 



(A.l) 



a</,^^(«)-«0o^(^) 



a,p,M^l{s)) - a^^{a^l{s)) 



A2{X) 



Ho{0',f>lis)) - a^^{a^l{s)) 



A 



and where we simply used that a(j,g{a, (s)) = s = a^^{ai (s)). Let us examinate 
separately the convergence of the two factors Ai and A2 in (A.l). From Step 2, we 
have 



hence 



lim a, is) = a. is) 



lim ^i(A) = —J = J . 

^^° "'^oKo(*)) A7^V2 {a-/{s)-<i)Q{x)f'^dx 



(A.2) 



Now, (3.4) and Step 2 imply: 

lim A2{X) = 47rV2 / {alHs) - (t>o{x)fi\{x)da 



(A.3) 



Therefore, (A.l), (A.2) and (A.3) give (3.5). This concludes the proof of Lemma 
[331 □ 



ORBITAL STABILITY OF SPHERICAL GALACTIC MODELS 35 



References 

[1] Aly J.-J., On the lowest energy state of a coUisionless self-gravitating system under phase 

volume constraints. MNRAS 241 (1989), 15. 
[2] Antonov, A. V., Remarks on the problem of stability in stellar dynamics. Soviet Astr., AJ., 4, 

859-867 (1961). 
[3] Antonov, A. V., Solution of the problem of stability of a stellar system with the Emden density 

law and spherical velocity distribution. J. Leningrad Univ. Se. Mekh. Astro. 7, 135-146 (1962). 
[4] Arsen'ev, A. A., Global existence of a weak solution of Vlasov's system of equations, U.S.S.R. 

Computational Math, and Math. Phys. 15 (1975), 131-141. 
[5] Batt, J.; Faltenbacher, W.; Horst, E., Stationary spherically symmetric models in stellar dy- 
namics, Arch. Rat. Mech. Anal. 93, 159-183 (1986). 
[6] Binney, J.; Tremaine, S., Galactic Dynamics, Princeton University Press, 1987. 
[7] Diperna, R. J.; Lions, P.-L., Global weak solutions of kinetic equations. Rend. Sem. Mat. Univ. 

Politec. Torino 46 (1988), no. 3, 259-288 (1990). 
[8] Diperna, R. J.; Lions, P.-L., Solutions globales d'equations du type Vlasov-Poisson, C. R. Acad. 

Sci. Paris Ser. I Math. 307 (1988), no. 12, 655-658. 
[9] Dolbeault, J.; Sanchez, O.; Soler, J., Asymptotic behaviour for the Vlasov-Poisson system in 

the stellar-dynamics case. Arch. Ration. Mech. Anal. 171 (2004), no. 3, 301-327. 
[10] Doremus, J. P.; Baumann, G.; Feix, M. R., Stability of a Self Gravitating System with Phase 

Space Density Function of Energy and Angular Momentum, Astronomy and Astrophysics 29 

(1973), 401. 



[11 
[12 
[13 



[14 

[15 

[16 

[17 

[18 
[19 

[20 

[21 

[22 

[23 

[24 

[25 

[26 

[27 

[28 



Fridmann, A. M.; Polyachenko, V. L., Physics of gravitating systems. Springer- Verlag, 1984. 

Gardner, C.S., Bound on the energy available from a plasma, Phys. Fluids 6, 1963, 839-840. 

Gillon, D.; Cantus, M.; Doremus, J. P.; Baumann, G., Stability of self-gravitating spherical 
systems in which phase space density is a function of energy and angular momentum, for 
spherical perturbations. Astronomy and Astrophysics 50 (1976), no. 3, 467-470. 

Guo, Y., Variational method for stable polytropic galaxies. Arch. Rat. Mech. Anal. 130 (1999), 
163-182. 

Guo, Y.; Lin, Z., Unstable and stable galaxy models, Comm. Math. Phys. 279 (2008), no. 3, 
789-813. 

Guo, Y.; Rein, G., Stable steady states in stellar dynamics. Arch. Rat. Mech. Anal. 147 
(1999), 225-243. 

Guo, Y.; Rein, G., Isotropic steady states in galactic dynamics, Comm. Math. Phys. 219 
(2001), 607-629. 

Guo, Y., On the generalized Antonov's stability criterion. Contemp. Math. 263, 85-107 (2000) 

Guo, Y.; Rein, G., A non-variational approach to nonlinear Stability in stellar dynamics 
applied to the King model, Comm. Math. Phys., 271, 489-509 (2007). 

Hormander, L, An Introduction to Complex Analysis in Several Variables (3rd Edition ed.), 
North-Holland, Amsterdam (1990). 

Hormander, L., L2 estimates and existence theorems for the d operator. Acta Math. 113 
(1965), 89-152. 

Horst, E.; Hunze, R., Weak solutions of the initial value problem for the unmodified nonlinear 
Vlasov equation. Math. Methods Appl. Sci. 6 (1984), no. 2, 262-279. 

lUner, R.; Neunzert, H., An existence theorem for the unmodified Vlasov equation. Math. 
Methods Appl. Sci. 1 (1979), no. 4, 530-544. 

Kandrup, H. E.; Sygnet, J. F., A simple proof of dynamical stability for a class of spherical 
clusters. Astrophys. J. 298 (1985), no. 1, part 1, 27-33. 

Kavian, O., Introduction a la theorie des points critiques et applications aux problemes ellip- 
tiques. Mathematiques & Applications (Berlin), 13. Springer- Verlag, Paris, 1993. 

Lemou, M.; Mehats, F.; Raphael, P., Orbital stability and singularity formation for Vlasov- 
Poisson systems. C. R. Math. Acad. Sci. Paris 341 (2005), no. 4, 269-274. 

Lemou, M.; Mehats, F.; Raphael, P., On the orbital stability of the ground states and the 
singularity formation for the gravitational Vlasov-Poisson system. Arch. Rat. Mech. Anal. 189 
(2008), no. 3, 425-468. 

Lemou, M.; Mehats, F,; Raphael, P., Stable ground states for the relativistic gravitational 
Vlasov-Poisson system, Comm. Partial Diff. Eq. 34 (2009), no. 7, 703-721. 



36 M. LEMOU, F. MEHATS, AND P. RAPHAEL 



[29] Lemou, M.; Mehats, F.; Raphael, P., A new variational approach to the stability of gravita- 
tional systems, submitted. 
[30] Lieb, E. H.; Loss, Analysis. Second edition. Graduate Studies in Mathematics, 14. American 

Mathematical Society, Providence, RI, 2001. 
]31] Lions, P.-L., The concentration-compactness principle in the calculus of variations. The locally 

compact case. I, Ann. Inst. H. Poincare Anal. Non Lineaire 1 (1984), no. 2, 109-145. 
]32] Lions, P.-L., The concentration-compactness principle in the calculus of variations. The locally 

compact case. II, Ann. Inst. H. Poincare Anal. Non Lineaire 1 (1984), no. 4, 223-283. 
]33] Lions, P.-L.; Perthame, B., Propagation of moments and regularity for the 3-dimensional 

Vlasov-Poisson system. Invent. Math. 105 (1991), no. 2. 
]34] Lynden-Bell, D., The Hartree-Fock exchange operator and the stability of galaxies, Mon. Not. 

R. Astr. Soc. 144, 1969, 189-217. 
]35] Mossino, J., Inegalites isoperimetriques et applications en physique. (French) ]Isoperimetric 

inequalities and applications to physics] Travaux en Cours. ]Works in Progress] Hermann, Paris, 

1984. 
]36] Marchioro, C; Pulvirenti, M., Some considerations on the nonlinear stability of stationary 

planar Euler flows, Comm. Math. Phys. 100 (1985), no. 3, 343-354. 
]37] Pfaffelmoser, K., Global classical solutions of the Vlasov-Poisson system in three dimensions 

for general initial data, J. Diff. Eq. 95 (1992), 281-303. 
]38] Sanchez, O.; Soler, J., Orbital stability for polytropic galaxies, Ann. Inst. H. Poincare Anal. 

Non Lineaire 23 (2006), no. 6, 781-802. 
]39] Schaeffer, J., Global existence of smooth solutions to the Vlasov-Poisson system in three 

dimensions, Comm. Part. Diff. Eq. 16 (1991), 1313-1335. 
j40] Schaeffer, J., Steady States in Galactic Dynamics, Arch. Rational, Mech. Anal. 172 (2004), 

1-19. 
]41] Sygnet, J.-F.; Des Forets, G.; Lachieze-Rey, M.; Pellat, R., Stability of gravitational systems 

and gravothermal catastrophe in astrophysics, Astrophys. J. 276 (1984), no. 2, 737-745. 
]42] Talenti, G., Elliptic equations and rearrangements. Ann. Scuola Norm. Sup. Pisa CI. Sci. 3 

(1976), no. 4, 697-718. 
]43] Wan, Y. H.; Pulvirenti, M., Nonlinear Stability of Circular Vortex Patches, Comm. Math. 

Phys. 99 (1985), 435-450. 
]44] Weinstein, M. I., Modulational stability of ground states of nonlinear Schrodinger equations, 

SIAM J. Math. Anal. 16 (1985), 472-491. 
j45] Wiechen, H., Ziegler, H.J., Schindler, K. Relaxation of collisionless self gravitating matter: 

the lowest energy state, Mon. Mot. R. ast. Soc (1988) 223, 623-646. 
]46] Wolansky, G., On nonlinear stability of polytropic galaxies. Ann. Inst. Henri Poincare, 16, 

15-48 (1999). 

CNRS AND IRMAR, Universite de Rennes 1, France 
E-mail address: mohaiiuned.lemou@univ-rennesl.fr 

IRMAR, Universite de Rennes 1, France 
E-mail address: florian.mehats@univ-rennesl.fr 

IMT, Universite Paul Sabatier, Toulouse, France 
E-mail address: pierre.raphael@math.univ-toulouse.fr 



